Jörn Alexander Quent MRC CBU
29 July 2020
\[t = \frac{\overline{x} - \mu}{s_\overline{x}}\] where \[ s_\overline{x} = \frac{s}{\sqrt{n}} \] where
\(\mu\) = Proposed constant for the population mean
\(\overline{x}\) = Sample mean
\(n\) = Sample size (i.e., number of observations)
\(s\) = Sample standard deviation
\(s_\overline{x}\) = Estimated standard error of the mean
Our toy example:
\[\underbrace{\frac{p(H_1 | data)}{p(H_0 | data)}}_\text{Posterior plausibility about hypotheses} = \underbrace{\frac{p(H_1)}{p(H_0)}}_\text{Prior plausibility about hypotheses} \times \underbrace{\frac{p(data| H_1)}{p(data| H_0)}}_\text{Bayes factor = Predictive updating factor}\]
Evidence in favour of alternative:
\[ BF_{10} = \frac{p(data| H_1)}{p(data| H_0)}\]
Evidence in favour of null: \[ BF_{01} = \frac{p(data| H_0)}{p(data| H_1)}\]
This mean that you can just inverse them: \[ BF_{10} = 1/BF_{01}\]
| \(BF_{10}\) | Evidence |
|---|---|
| > 100 | Extreme evidence for \(H_1\) |
| 30 – 100 | Very strong evidence for \(H_1\) |
| 10 – 30 | Strong evidence for \(H_1\) |
| 3 – 10 | Moderate evidence for \(H_1\) |
| 1 – 3 | Anecdotal evidence for \(H_1\) |
| 1 | No evidence |
| 1 – 1/3 | Anecdotal evidence for \(H_0\) |
| 1/3 – 1/10 | Moderate evidence for \(H_0\) |
| 1/10 – 1/30 | Strong evidence for \(H_0\) |
| 1/30 – 1/100 | Very strong evidence for \(H_0\) |
| < 1/100 | Extreme evidence for \(H_0\) |
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 3.236673 ±0%
##
## Against denominator:
## Null, mu = 0
## ---
## Bayes factor type: BFoneSample, JZS
ttestBF) assume the following priors:\[s_\overline{x} = \frac{s}{\sqrt{n}} \;\;\;\;\;\;\;\; t = \frac{\overline{x} - \mu}{s_\overline{x}}\]
This shows that a large sample size is necessary to a probability of 80% that \(BF_{10}\) > 10 or \(BF_{10}\) < 1/6.
| Effect size | Necessary sample size | Misleading evidence in % | £ for online experiment | £ for fMRI experiment |
|---|---|---|---|---|
| 0.5 | 72 | 0.0003 | 605 | 39600 |
| 0.0 | 232 | 0.0011 | 1949 | 127600 |
| Effect size | Average sample size | Maximal sample size | Misleading evidence in % |
|---|---|---|---|
| 0.5 | 41 | 170 | 0.13 |
| 0.0 | 83 | 2765 | 2.95 |
Costs for traditional designs:
| Effect size | Necessary sample size | Misleading evidence in % | £ for online experiment | £ for fMRI experiment |
|---|---|---|---|---|
| 0.5 | 72 | 0.0003 | 605 | 39600 |
| 0.0 | 232 | 0.0011 | 1949 | 127600 |
Costs for pure sequential design:
| Effect size | Average sample size | Maximal sample size | Misleading evidence in % | £ for online experiment | £ for fMRI experiment |
|---|---|---|---|---|---|
| 0.5 | 41 | 170 | 0.13 | 344 | 22550 |
| 0.0 | 83 | 2765 | 2.95 | 697 | 45650 |
| Effect size | Average sample size | Strong evidence in % | Misleading evidence in % | Insufficient evidence in % |
|---|---|---|---|---|
| 0.5 | 39 | 98.05 | 0.12 | 1.83 |
| 0.0 | 58 | 80.25 | 2.31 | 17.44 |
But
All code can be found here. Further points and more simulations can be found here.